How have these demographics changed over time?

While overall percentages are informative, we can also examine trends in representation of these different student groups across course terms.

The below plot shows the percent of students in each term of the course of interest across the range of terms in the dataset. Student identities are shown in different colored lines over time. Individual terms (winter, spring, summer or fall) are labeled on the x-axis.

What majors do students in this course hold?

In addition to demographic factors, student majors also provide information about students’ course background, interest, and potentially familiarity with the material in our courses.

The below plot shows all students across all terms, broken down by major of study. Each bar shows the percent of students in each major. The color of the bar indicates whether the major is a STEM major, or a non-STEM major. (The overall percent of STEM majors in the course is shown in the bar plot above with the other student identities.)

Additionally, we may only be interested in a few majors, such as the major of the department of the course. If specified, the plot below shows the percent of majors of interest in this course, compared to all other majors.

What is the grade distribution in this course?

The below plot shows the overall grade distribution for this course across all terms. Note that this plot excludes students who did not take the course for a lettergrade (pass/non pass) or withdrew.

How do course outcomes compare for different student groups?

The overall grade distribution gives us a sense for grade patterns in this course. The overall grade distribution gives us a sense for grade patterns in this course. However, from an equity perspective, we want to examine and compare the distributions for different student identities.

The below plot shows the full distribution of the grades for multiple student identities, both academic (STEM major, transfer students) and demographic (PEER, women, low income, first-generation, international).

• The distribution is shown as a density plot with the majority (25th - 75th percentiles) shaded in a darker color. This region represents the “box” portion of a box plot.

• The mean and the 95% confidence interval around the mean is shown as a point and errorbar below each distribution.

  • If 95% confidence intervals do not overlap between two groups, the means are statistically significantly different.

• The medians is shown as a small vertical line on each distribution.

• The overall mean grade across all students (across all terms in the dataset) is shown as a dashed vertical line.

While viewing the full distribution of grades allows us to view the full range, sometimes comparing average grades, especially between majority and historically-excluded / minoritized groups can be informative when evaluating equity in course outcomes.

The below plot shows the overall average grades for students, disaggregated by whether they hold a certain identity or not.The x-axis range has been zoomed in so we can better evaluate differences between groups.

In this plot:

• Means and 95% confidence intervals are shown as points and errorbars.

• Student identities are represented by shape; triangles represent students in the historically excluded/minoritized group and circles are students in the majoirty/ historically supported/accepted group.

• The overall average grade for all students in all terms of the course is shown as a dashed line.

Note that it is more useful to compare between students who do and do not hold each identity than across groups, as the student identities are not mutually exclusive (a single student may belong to any and all of these identities).

As discussed previously, the “PEER” category can mask the nuanced different experiences of students of specific ethnic or racial groups. To explore the differences in course grades, mean course grades by ethnicities are disaggregted below.

In this plot: * Means and 95% confidence intervals are shown as points and errorbars.

• The overall average grade for all students in all terms of the course is shown as a dashed line.

How have grades across student groups changed over time?

We can also evaluate how grading and grade distributions have changed over time in this course.

Below, we can see general trends in grade distribution over time, where the distribution of grades is shown as a density plot. The color of the line represents the term.

The below plot shows the trends over time for each student identity.

• Each axis of identity is a separate plot.

• Means and 95% confidence intervals are shown as points and vertical errorbars.

• The majoritarian group (i.e., students who do not hold the indicated identity) is shown in gray circles across all plots. Students who do hold the identity are shown in colored triangles.

*The overall course average, across all students and all terms, is shown with a dashed line.

How does overall prior performance at the university (i.e., GPAO, prior GPA) compare to student outcomes in my course?

One way we can compare outcomes in our course to students’ prior performance at our institution is by calculating grade anomaly. Grade anomaly subtracts a measure of student’s general or prior academic performance, such as prior GPA, from their course grade.

SEISMIC projects often use grade point average omitting the course of interest (GPAO), because this metric has been found to be the best metric of prior academic preparation in previous studies (compared to high school GPA or standardized test scores, Koester et al.,2016). GPAO can also be especially useful for transfer students, who may not have a prior GPA at our institution if they just transferred.

Grade anomalies can be:

• Grade penalties - where students receive lower grades in the course relative to their other coursework, or,

• Grade bonuses - where students receive higher grades relative to other courses.

Whether a course confers a grade bonus or a grade penalty depends on what other courses students have taken, or take in the term of interest. In many ways, the grade anomaly depends on where the course is situated in the curricular context (i.e., is this course taken along wiht many other large introductory STEM courses, or are the other courses taken to this point general education or electives?).

Importantly, grade bonuses or penalties are not necessarily “good” or “bad”, and it often depends on the goals of the course and the assessment structure. For example, courses that aim to evaluate students’ level of mastery of specific skills or concepts as prerequisites for future coursework (a goal of many introductory STEM courses) tend to confer grade penalties than other courses.

First, we can explore what the distribution of course grades are versus GPAO. The plot below shows the sample size of students for different combinations of GPAO versus course grade. The line shows the one-to-one line, so areas below the line represent grade penalties (GPAO > grade), areas above grade bonuses (GPAO < grade). Lighter colors represent more students that fall in that quadrant.

As with actual grades, we can also examine grade anomaly across student demographics. The following plot shows the distribution of grade anomalies for each listed student demographic. The darker portion of the distribution represents the students between the 25th and 75th percentile. The mean is a circle, with errorbars showing the 95% confidence interval around the mean. Vertical lines on each plot represent the median. The dashed line across the whole plot represents the mean grade anomaly across the entire coures (all terms and all students).

The below plot focuses on the mean grade anomaly for each of the student demographics, comparing the majority group (circles) to the historically excluded / marginalized group (triangles). Points show means, flanked by the 95% confidence interval for each group.

Similarly, we can also further disaggregate the PEER group into specific student races/ethnicities. The below plot shows grade anomalies by student ethnicity. Points represent mean grade anomaly, error bars the 95% confidence interval.

How does grade anomaly change over time?

We can also observe these trends over time. The below plot shows the trends in average GPAO and course grade over the terms listed in the dataset. The vertical distance between these points shows the size of the grade anomaly, on average, for that term. If the course grade point is above the GPAO, there is a grade bonus, if below, a grade penalty.

Then, we can show trends in grade anomaly for each student demographic over time. Points represent mean grade anomaly for all students of that identity in that term, and errorbars are the 95% confidence interval. The majoritarian group (i.e., students who do not hold the indicated identity) is shown in gray across all plots. The overall course average grade anomaly, across all students and all terms, is shown with a dashed line.

How do course outcomes differ across the spectrum of systemic advantages students may have access to in higher education?

How can we view the cumulative effect of systemic advantages students may have access to (conferred by demographic identities) overall in the context of a course?

Acknowledgements

This report is heavily inspired by the Foundational Course Initiative reports from the University of Michigan, created by Dr. Heather Rypkema and refined by the Assessment Toolkit team. The R code in this report was initially drafted by Victoria Farrar (UC Davis) with feedback from Nita Tarchinski (Michigan), Matthew Steinwachs (UC Davis), Jenna Thomas (UC Davis), Tim McKay (Michigan), Heather Rypkema (Michigan), Marco Molinaro (University of Maryland) and Stefano Fiorini (Indiana University).